The present invention relates to the field of image registration, and more particularly to an image registration system implementing PID control techniques.
In general, registration may be defined as the process by which the correspondence between two or more arrays of values is performed. Image registration may be defined as the process by which the correspondence between a subset or all points in two or more images, typically of the same scene, is determined. Image registration may also be stated as the process of estimating the affine transformation between the images. The affine transformation may be defined as the rotation, scale change, and translation that maps one n-dimensional image into a second n-dimensional image. In practice, n is usually 2 or 3. Image registration usually plays an important and critical first step in applications involving the fusion of information from multiple modalities. Modalities refer to various sensor implementations to receive the images or array of values. For example, an input image may be received by a thermal camera; whereas, a reference image may be received by a regular camera. An input image may exhibit an affine transformation with respect to the reference image.
Image registration may be used in image analysis tasks such as motion or change detection, fusion of data from multiple sensing modalities, and image geometric correction. There has been a tremendous increase in the need for good image registration techniques due to the increased use of temporal and multimodal 2-D and 3-D images in medical, remote sensing, and industrial applications.
Prior image registration techniques may have implemented some notions of feedback control systems. However, prior image registration techniques did not use PID (P-proportional, I-integration, D-differential components) control techniques to determine the affine transformation of one image with respect to another image.
Background of classical control systems is deemed appropriate. Control systems at the most basic level deal with a system that has a specific input which generates a specific output. The system could be a plant, an engine, a biological object, or any other structure with a clear inputxe2x80x94output behavior. Inputs to the control system may be interpreted as control actions or noise sources. Outputs of a given control system mainly comprise measurements. These measurements describe internal states, which, in most cases, are not directly measurable.
In the simplest case, a so-called set point must be reached and stabilized based on appropriate control actions. Even in highly nonlinear situations, linear models can successfully be used to determine such control strategies. For example, the general continuous linear model                                                                                           ⅆ                                      x                    ⁡                                          (                      t                      )                                                                                        ⅆ                  t                                            =                              xe2x80x83                            ⁢                                                Ax                  ⁡                                      (                    t                    )                                                  +                                  Bu                  ⁡                                      (                    t                    )                                                                                                                                          y                ⁡                                  (                  t                  )                                            =                              xe2x80x83                            ⁢                                                Cx                  ⁡                                      (                    t                    )                                                  +                                  Du                  ⁡                                      (                    t                    )                                                                                                          (1)            
can be described by:
matrices A, B, C, and D
vector x(t) of internal states
vector u(t) of control actions
vector y(t) of measurements.
Additionally, noise terms can be added if necessary. The most successful and very common control strategy is simply a feedback according to:
u(t)=xe2x88x92K(t)x(t)xe2x80x83xe2x80x83(2)
The design of a valid control system mainly comprises the construction of matrices K(t) making the control system as efficient as possible. In typical applications the matrix K(t) is constant which simplifies the algorithms considerably.
Prior image registration techniques implementing notions of feedback strategies were deficient in that the convergence rate to determine the affine transformation was unsatisfactory. Therefore, an improved system and method is desired for performing registration, e.g. determining the affine transformation of an input image with respect to a reference image.
A system and method are provided for improving the accuracy and convergence rate in determining the affine transformation of one image or array of values with respect to another image or array of values. According to one embodiment of the present invention, the problem of finding the affine transformation parameters between the images or array of values to be registered is posed as a classical control problem using PID (P-proportional, I-integration, D-differential components) techniques. According to one embodiment, registration may be performed between an input array of values and a reference array of values. According to another embodiment, image registration may be performed between an input image and a reference image. An efficient registration or image registration may be performed by estimating the transformation between two subsets using a strategy based on a PID control system.
The present invention comprises a system, method and memory medium for improved registration, such as image registration. In a first embodiment, the system comprises a PID (P-proportional, I-integration, D-differential components) controller to accurately determine the affine transformation. Hence the PID controller may be used to determine the amount of rotation, shift and scale of an input image relative to a reference image. Both the reference and input image may be comprised of pixels and may be functions of position, angle and scale. In another embodiment, the PID controller may be used to determine the amount of rotation, shift and scale of an input array of values relative to a reference array of values. The reference and input array of values may both be functions of position, angle and scale.
In a particular embodiment, the reference image may be preprocessed by constructing a representation of the reference image, preferably a gradient matrix, xcex, comprising gradient information of the position values of the reference image. The input image may be input to an image registration system.
An estimate may then be constructed comprising initial estimates of at least one of a position, angle and scale value of the input image. Preferably, the estimate is an estimate matrix, xcfx81.
The input image may then be compared with the reference image to determine if the input image has been shifted, rotated or scaled with respect to the reference image. More specifically, an error may then be calculated by subtracting the input image from the reference image. The error is preferably represented by an error matrix, e.
A comparison may subsequently be made between the error matrix and a given value. If the error matrix is less than a given value, then the correspondence between the input image and the reference image has been determined. Otherwise, the process continues by multiplying the error matrix, e, with the gradient matrix, xcex, and the estimate matrix, xcfx81. The result of the multiplication is a new change in the position, angle and scale values of the input image. The input image is consequently transformed by replacing the old values of the position, angle and scale with the new values of the position, angle and scale. These new values are constructed from the changes in the position, angle and scale values of the input image. For example,
Xnew=X+K*xcex94X+Ki*∫xcex94X+Kd*d/dx(xcex94X)
Ynew=Y+K*xcex94Y+Ki*∫xcex94Y+Kd*d/dy(xcex94Y)
angle new=angle+K*xcex94angle+Ki*∫xcex94angle+Kd*d/dangle(xcex94angle)
scale new=scale+K*xcex94scale+Ki*∫xcex94scale+Kd*d/dscale(xcex94scale).
The integral constant, Ki, is a steady state error correction parameter that is multiplied to the average of the previous values of the change in position, angle and scale. The derivative constant, Kd, is preferably a response parameter which may reduce the swings in the estimate values from being too high or too low. K is a constant that is multiplied to xcex94X, xcex94Y, xcex94 angle, and xcex94 scale. xcex94X is the change in the value of X of the input image. xcex94Y is the change in the value of Y of the input image. xcex94 angle is the change in the angle value of the input image. xcex94 scale is the change in the scale value of the input image.
The input image is transformed by replacing the old values of the position, angle and scale in the input image, with the new values. The transformed input image may then be subtracted from the reference image, providing a new error matrix, e. The estimate matrix, xcfx81, may also be updated using the error matrix, e, the current version of the estimate matrix, xcfx81, and the gradient matrix, xcex. The process described is then repeated one or more times until the error matrix, e, is less than a given value. One the error matrix, e, is less than the given value, the affine transformation is determined to have been found.
To improve the convergence of finding the actual position, angle and scale values of the input image, appropriately valued constants, K, Ki and Kd are used. Appropriate values of K, Ki and Kd may be found by first receiving initial estimates of K, Ki and Kd. The reference image may then be rotated, shifted and scaled, producing a pseudo input image using a known affine transformation. The above registration method may then be used using the pseudo image as the above mentioned input image. Consequently, image registration may be performed between the reference image and the pseudo input image using PID control techniques as discussed above. Once the error matrix, e, is less than a given value, then appropriate values for K, Ki and Kd have been found for that iteration. These K, Ki and Kd values are known as candidate values.
After candidate values have been found, the reference image may be shifted, rotated and scaled to create a new pseudo image if the convergence rate of finding the actual position, angle and scale values of the pseudo image is unsatisfactory. Image registration may be performed again between the reference image and the new pseudo image using the candidate K, Ki and Kd values from the previously produced pseudo image. Image registration may also be performed using new initial values for K, Ki and Kd. Image registration between the reference image and the newly created pseudo images may be repeated one or more times until the convergence rate of finding the actual position, angle and scale values of the pseudo image is satisfactory.
Final K, Ki and Kd values result once the convergence rate of finding the actual position, angle and scale values of the pseudo image is satisfactory. Final K, Ki and Kd values may be produced after continually adjusting the candidate values of K, Ki and Kd from each iteration or a subset of iterations. Final K, Ki and Kd values may also be produced by taking an average of the candidate K, Ki and Kd values of each iteration instead of continually adjusting the values of K, Ki and Kd from prior candidate K, Ki and Kd values. The final K, Ki and Kd values are used in transforming the input image to approximate the reference image.
In a particular embodiment, a method for performing image registration between an input image and a reference image comprises a step of receiving the input image. The reference image comprises a first plurality of pixels. The input image comprises a second plurality of pixels, where the input image differs from the reference image by one or more of a position, angle and scale. The method further comprises the step of transforming the input image to approximate the reference image, where the transforming uses PID control techniques. Furthermore, transforming the input image produces a new input image. The method further comprises the step of calculating a difference between the new input image and the reference image resulting in an error. The steps of transforming the input image and calculating the difference are performed one or more times until the error is less than a given value. The method further comprises the step of generating an estimate in response to transforming and calculating the difference. The estimate comprises at least one of a position value, an angle value and a scale value which defines an affine transformation of the input image with respect to the reference image.